Welcome back to pattern recognition. So today we want to talk a bit about norms and we will

look into different variants of vector norms, matrix norms and the like and have a couple

of examples how they are actually applied.

So let's look into norms and later on also norm dependent linear regression. So we've seen that

norms and similarity measures play an important role in machine learning and pattern recognition.

In this chapter we want to summarize important definitions and facts on norms and I think this

is also a very good refresher. If you're not so familiar with norms this video will be a very

good summary of the most important facts on norms. So we'll consider the problem of linear

regression then later in the next video for different norms but in this video we'll mainly

talk about norms and their properties. So later we also look briefly into the associated optimization

problems. So let's start with the inner product, the inner product of two vectors that we already

used in this lecture. We write this up as x transpose y and this essentially is the sum over

the element wise multiplication. This can then be used to define the Euclidean L2 norm. So this is

essentially the inner product of the vector with itself and then we take the square root. So we can

write this as x transpose x square root or you could say that's element wise the square summed

up and then taken the square root. Also we can define inner products of matrices and if we want

to do that then we see that the inner product of a matrix is essentially x transpose y and then we

take the trace of the resulting matrix and this then results in a sum over all the elements of x

multiplied with all the elements of y and then we compute the essentially two-dimensional sum

here because they're matrices but we mix all the elements with each other and sum them up. Now if

you follow this definition then you can find the Frobenius norm as the inner product of x with

itself so this means that we essentially square all of the elements and then sum them up. So this

is the Frobenius norm. So what else do we have? Well what's a norm actually? The function that we

indicate with these double bars is called a norm if it is non-negative so the result of the norm is

always zero or greater to zero for all given x then it should be the finite which means that it

will be zero only if all of the entries of x are zero and it should be homogeneous which means that

if you multiply a with x and a is a scalar value and you take the norm then this is the same as

taking the absolute value of a and multiplying it with the norm of x and this is of course true

for scalar values. Then it should also fulfill the triangle inequality which means that for all x and

y if you have some x and you add y and take the norm then this should be lesser or equal to the

norm of x plus the norm of y also known as the triangle inequality. Then we can also find other

norms for example the L0 norm which is essentially denoting the number of non-zero entries despite

its name it's not really a norm so we will see that and you see that often used also in rigid

the L0 norm and also other variants that are actually not norms but they're used in this

context. So generally this will allow us to define an LP norm and now P is a scalar value and note

that this is small caps P and it's defined as norm with P greater or equal to one of a d-dimensional

vector and then you can compute this essentially with the element wise x i's to the absolute value

to the power of P then summed up and then you take the Pth root of the respective sum. Technically

you can also do this with values lower than one but then you end up with constellations that no

longer fulfill all of the previous norm properties. In literature you can also find this as a 0 norm

or as L0.5 norm and so on. So you essentially use this same idea of computing them but you use values

that are smaller than one for this. Now let's look into real norms so there's the L1 norm which is

essentially the sum of absolute values then there's the L2 norm that is essentially the sum of squares

and the square root and then there's also other norms following this direction up to the L infinity

norm and this is also called maximum norm and here you essentially pick the maximum absolute value

that occurs in the vector and return it as the respective result of the norm. So there's also

LP norms but now P is a capital P and bold so this is a matrix and if it is a symmetric positive

definite matrix then the quadratic LP norm can be given in the following way so this is essentially

x transpose times P times x so it's essentially a weighted distance to itself weighted by P and

then you take the square root. So you can also write this up in the following way if you split